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ABSTRACT 

The Compton Gamma Ray Observatory was the 
second of NASA s Great Observatories. At 17^2 tons, it 
was the heaviest astrophysical payload ever flown at the 
time of its launch on April 5, 1991 aboard the Space 
Shuttle. During initial, on-orbit priming of the 

spacecraft s monopropellant hydrazine propulsion system, 
a severe waterhammer transient was experienced. At that 
time, anomalous telemetry readings were received from 
on-board propulsion system instrumentation. This led to 
ground analyses and laboratory investigations as to the 
root cause of the waterhammer, potential damage to 
system integrity and functionality, and risks for switching 
from the primary (A-side) propulsion system to the 
redundant (B-side) system. The switchover to B-side was 
ultimately performed successfully and the spacecraft 
completed its basic and extended missions in this 
configuration. Nine years later, following a critical 
control gyroscope failure, Compton was safely deorbited 
and re-entered the Earth’s atmosphere on June 4, 2000. 
Additional risk assessments concerning viability of A- 
and B-sides were necessary to provide confidence in 
attitude and delta-V authority and reliability to manage 
the precisely controlled reentry. This paper summarizes 
the design and operation of the propulsion system used on 
the spacecraft and provides ‘lessons learned” from the 
system engineering, investigations into the propellant 
loading procedures, the initial priming anomaly, mission 
operations, and the commanded re-entry following the 
gyro failure. 

INTRODUCTION 

The Compton Gamma Ray Observatory (CGRO) was 
a large scientific spacecraft designed for celestial 
observations from low Earth orbit. It was the second 
element launched in NASA’s deployment of four “Great 
Observatories” (HST, CGRO. Chandra-AXAF. SIRTF) 
and carried instruments dedicated to the highest part of 
the electromagnetic spectrum. The objective of the CGRO 
mission was to obtain gamma-ray measurements over the 
entire celestial sphere with unprecedented sensitivity, 
spectral range and resolution. 

CGRO was launched aboard the Space Shuttle 
Atlantis (STS-37) on April 5. 1991, and was deployed 
April 7 into a 45” km circular orbit at 2S.5 degrees 
inclination (Figure l l At the time of its deployment it set 
two records for non-military spacecraft: it was the largest 
spacecraft launched by STS and it had the largest 
monopropellunt propulsion system ever flown. 
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Figure 1. CGRO Deployment from STS-37 


After nine years of exciting scientific discoveries 1 “ of 
very energetic celestial phenomena (far exceeding its 
minimum mission life of 27 months), CGRO was safely 
de-orbited with controlled re-entry into the Earth’s 
atmosphere on June 4. 2000. Pieces of the spacecraft 
survived the re-entry, landing in a remote part of the 
Pacific Ocean near the equator, approximately 3.S62 km 
(2.400 miles) southeast of Hawaii. 

The CGRO mission was a NASA cooperative 
program managed by the NASA Goddard Space Flight 
Center (GSFC) and included co-investigators from the 
United States, Federal Republic of Germany . Netherlands. 
ESA and United Kingdom. The Observatory carried four 
highly sophisticated instruments capable of making 
simultaneous measurements over six decades of enersv 
(20 keV-30 GeV). These instruments were: the Burst 
and Transient Source Experiment (BATSE). the Oriented 
Scintillation Spectrometer Experiment (OSSE), the 
Imaging Compton Telescope (COMPTEL). and the 
Energetic Gamma Ray Experiment Telescope < EGRET). 

The CGRO spacecraft was designed and developed 
by TRW in Redondo Beach. CA. Table i presents a 
summary of the spacecraft subsystems. The 
monopropellant hydrazine propulsion subsvstem 
consisted of “A-side” and fully redundant “B-side” sets of 
thrusters, teed system components and propellant tanks. 
Although normally inactive during science uathennii, the 
propulsion system was to be used every 2^ vears to 
reboost CGRO to offset decay in orbital altitude due to 
atmospheric drag. 

The spacecraft :> propui.sion subsystem had two major 
on-orbit anomalies during :he mission. The fir>: anomaly 
■ occurred during checkout and activation of the spacecraft 
arter being released from the Shuttle, (mmediateiv upon 
opening one of the propellant tank isolation valves n the 
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Table l. CGRO Spacecraft Subsystems 


Science Instruments 

• Four main mstruments \ BA FSE. OSSE. COM PTE L. 

EGRET v comprising approximately 6.300 kg (7 tonsi 

• Continuous detection over 20 Kcv to 30 Gev range 

Structure (bolted aluminum box-girder framework) 

• Mass i loaded): 1 5.876 kg < 35.000 Ibm) 

• Body Size: 4.6 m \ 5.5m \ 0. 1 m (2 1.3 m SA span) 

Power Subsystem 

• Solar Array Power: two accordion-stvle. deployable 

arrays generating 4300 w BOU3980 w EOL with 
396 ft : area 

• Battery Power: six Ni-Cd battenes at 50 A-hr each 

Thermal Subsystem 

• Uses coatings, blankets, louvers, radiators and heaters 

• Science instruments thermally isolated from spacecraft 

and each other 

• Redundant thermostats and heater elements 

Communications & Data Handling Subsystem 

• Standard NASA modular design based on Solar Max 

and Landsat 4 & 5 spacecraft 

• S-band telecom using 1.52 m (60 inch) HGA 

• Two omnidirectional LGAs 

• Two second generation TDRSS transponders 

• Uplink at .125 or 1.0 Kbps: downlink at 32 Kbps (256- 

512 Kbps via TDRSS) 

• Two NASA standard tape recorders for playback at up 

to 5 12 Kbps via HGA and TDRSS 

• Advanced clock for time accuracy to .0001 second 

Attitude Control & Determination Subsystem 

• 3-xxis stabilized, zero momentum biased control system 

using reaction wheels with magnetic unloading 

• attitude sensors 

o Fixed head star trackers (3) 

o Inertial reference gyros (4) 

o Coarse Sc fine sun sensors 

• Attitude control 

o Reaction wheel assemblies (4) 
o Monopropellant rocket thrusters (8) 

• Sinale target pointing control for up to 14 days 

• Pointing control to ±0.5°; measurement to ±0.03° 

Propulsion Subsystem 

• Hydrazine propellant: 1924 kg 1,4240 Ibm). High Purity 

• GN2 pressurant: 17.2 kg (38 Ibm) 

• Four 440 N i 100 Ibf ) Orbit Adjust Thrusters 

• Eight 22 N < 5 Ibf} Attitude Control Thrusters 

• "Blowdown” operating pressure: 2760 kPa (400 psia) 

BOL to 600 kPa {87 psia) EOL 

• Four titanium propellant tanks, each with AFE-332 

diaphragm 

• On-orbit refueling module 

• Safety compliant with NHB 1700.7A 


A-Nide propellant manifold, telemetry indicated that two 
of the isolation valves had uncommarJcd changes in the 
open/close position statu> and one of the pressure 
transducers indicated “over-limit** pressure. Since the 
science activities were not affected, re-activation ot the 
propulsion system was postponed until April 1993 at 
which time the Observatory orbit had decayed to 350 km. 

This two-year interval allowed a comprehensive 
analysis of the anomaly and detailed development of 
corrective actions. Ground tests and analyses concluded 
that, in spite of being “fully** loaded with hydrazine, the 
A-side propellant manifold had been exposed to very high 
surge pressures (a “waterhammer" transient). To prevent 
reoccurrence, a method was successfully developed to 
safely prime the B-side manifold by opening the isolation 
valves for very short durations to slowly raise the 
hydrazine pressure in the downstream manifold to design 
operating pressure. The B-side Attitude Control Thruster 
(ACT) manifold was successfully primed Aprii-Juiy 1993 
in preparation for restoring the orbital altitude of CGRO 
to 450 km. 

The second propulsion subsystem on-orbit anomaly 
occurred during the calibration burn segment of the orbit 
raising operation. The plan was to raise orbit using only 
the four B-side ACTs. During the test burns, one of the 
ACTs (designated “ACT-B2") produced unacceptably 
low thrust. Ground tests determined that the low thrust 
was most likely related to flexing of the thruster valve 
seal at high propellant mass flow rate. 

To compensate for the low thrust, two of the four 
Orbit Adjust Thrusters (OATs) would be needed to 
maintain attitude control during the orbit raising bums. 
The B-side OAT manifold was successfully pressurized 
using procedures previously employed to prime the B- 
side ACT manifold. 

The first orbit reboost of the CGRO was completed in 
December 1993 and restored the orbit to 450 km circular 
from a low point of 350 km. As predicted, the 
performance of ACT-B2 became nominal as the operating 
flow rate decreased, a natural result of the propellant tank 
pressure decay in blowdown mode. A second orbit 
reboost was performed in March-June 1997 with nominal 
performance on all B-side ACTs and OATs. 

The propulsion subsystem performed flawlessly 
during four critical, controlled re-entry burns that ended 
the CGRO mission on June 4, 2000. 

OVERVIEW OF PROPULSION SUBSYSTEM 

Figure 2 shows the propulsion subsystem complete 
on a buildup fixture prior to transfer to the spacecraft 
structure. Figure 3 is a schematic of this subsystem. 
CGRO was the first scientific spacecraft designed for on- 
orbit refueling of propellant. The on-orbit refueling 
module contained a NASA-supplied propellant coupling. 
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Figure 2. CGRO Flight Propulsion Subsystem 

The propulsion subsystem consisted of A-side and 
fully redundant B-side sets of thrusters, feed system 
components and propellant tanks. As will be seen, it is 
significant that this design featured functional redundancy 
with fault isolation capability. Crossover isolation valves 
permitted full utilization of propellant and provided 
capability for center-of-mass management by control of 
the quantity of propellant used from each tank. Details of 
the design and development of the CGRO propulsion 
subsystem were given in a previous paper 3 . 

The mission tasks for the propulsion subsystem were 
to provide: 

• Orbital altitude restoration (drag make up) 

• Attitude control during reboost 

• Descent for refueling and on-orbit servicing 

• Ascent (from STS servicing orbits) 

• Descent for STS retrieval or controlled reentry 

• Provide safe hold operating mode in event of loss 

of gyroscope stabilization. 

The system was designed to operate in a pressure 
blowdown mode over a range of 400 to 87 psia. The four 
440 N OATs were to be tired simultaneously to provide 

impulse tor orbit altitude change, orbit maintenance, 
descent tor refueling, ascent and controlled reentry. The 
OATs were placed on the spacecraft X and V axes with 
thrust vectors parallel to the Z-axis (see Figure 2). The 
OATs were to be oft-modulated to provide primary 
attitude control about the spacecraft pitch and roll axes 
during the AV firings. The 22 N ACTs were to provide 
primary yaw attitude control during operation of the 
OATs. In the e\ent that one of the OATs failed during 
tiring, its geometric opposite would be automatically shut 
down and impulse would continue to be provided by the 
remaining OAT pair 

The ACT') were canted >ff the spacecraft Z-axis and. 
wncn tired appropriately in pairs, could provide control 


Figure 3, CGRO Propulsion Subsystem Schematic 

torques about any of the three spacecraft axes. They 
provided primary yaw control and secondary pitch and 
roll control during OAT Firings. Thrust levels and 
moment arms about the spacecraft center-of-mass were 
such that the ACTs could provide complete three-axis 
control at all times. They were to operate primarily in a 
pulse mode, but were designed for steady-state 
operation — a design feature that was used when thev were 
required to backup the OATs for altitude raising. 

Of particular significance to following discussions, 
the subsystem employed multiple latching isolation valves 
(indicated in Figure 3) to direct and lock off propellant 
flow. The design ot the isolation valve is shown in Figure 
4. It is a pressure-balanced, dual coil, solenoid-operated 
latching shutoff valve with downstream backpressure 
relief capability. The design features a closure spring to 
hold the valve closed and a permanent magnet latch to 
hold the valve open without continuous power drain. The 
inner solenoid coil is powered to open the valve and the 
outer coil is powered to close the valve. The critical 



Figure 4. CGRO Latching Isolation Valve Design 
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^lll!,la , of the poppet is spherically shaped to close on a 
tetlon ring seal. The solenoid and trim (closure) spring 
are hermetically- sealed from propellant by welded 
bellows assemblies. The spring assembly controlling the 
backpressure relief function is exposed to the fluid. A 
100-micron absolute wire mesh filter is installed in each 
of the fluid ports. A position switch assembly, which 
electrically indicates valve position, is located at the top 
and is integral with the valve. Other than being damaged 
by above-specification overpressures during the A-side 
priming attempt, the latching isolation valves performed 
nominally during the nine years of on-orbit life. 
Moreover, it was this valve's beyond-specification 
capability to respond to millisecond-level pulse 
commands that ultimately provided a means to safely 
prime the subsystem B-side, thereby enabling the CGRO 
mission to continue with low risk. 

PROPELLANT LOADING SEQUENCE 
AND ISSUES 

The propulsion system was designed for a STS 
launch, incorporating two-fault tolerance for propellant 
leakage and planning to launch ‘wet" to the thruster 
valves but unpressurized (at atmospheric pressure) 
downstream of the tank isolation valves Al, A2, Bi and 
B2. 

In conducting normal leakage tests using nitrogen 
gas, the leakage rates of isolation valve A2 and crossover 
isolation valve B1A were determined to exceed 
specification. The valve leakage problems occurred late 
in the program, resulting in a Noncompliance Report. A 
condition for approving CGRO for launch was 
"demonstration of no continuous liquid leakage" in the 
launch configuration. This demonstration was 
accomplished by a partial fueling operation in June 1990 
at which time 15 Ibm of hydrazine was loaded into the A- 
side of the system. The system was monitored for liquid 
leakage; none was observed and therefore launch was 
approved. 

The partial fueling contributed to the surge problem 
because a different propellant loading system was used 
for this small mass of propellant. 

The loading operation was as follows. Tank Al fuel 
and gas fill and drain valves and isolation valves Al. A4, 
and A3 were opened. The tank was vented and evacuated 
to 26 inches of mercury (Hg). The gas fill and drain valve 
was closed and the fuel side evacuated to "30 inches of 
mercury" (gauge resolution). 15.7 Ibm of hydrazine was 
loaded and the fuel fill and drain valve (FDV) closed. 
The gas FDV was then opened and pressurized to 15.1 
psia. Isolation valves Al. A3 and A4 were then dosed 
and the tanks pressurized to 430 psia. The system was 
monitored for leakage for 5 days before depressurizing 
the A-side tanks to 25 psia. 

The importance of the above operation is that the 
"partial fueling" operation utilized a hydrazine supply 


consisting of a -.mail tank pressurized to N) psig •O'' psia). 
This tank's storage time and handling were sufficient for 
the propellant to become fully saturated with nitrogen gas 
at 65 psia. Nitrogen gas bubbles which eventually formed 
in the propellant feed lines came from two sources: 
residual nitrogen in the system prior to filling (imperfect 
vacuum), and nitrogen that came out of solution from the 
hydrazine due to depressurizing the lines to 25 psia until 
launch and deployment. 

The system vacuum before loading hydrazine was 
estimated at 10 torr (gauge read "30 inches " mercury). 
This yields a volume ratio of 1.3*7 when re-pressurized to 
14.7 psia. The volume of nitrogen coming out of solution 
from the hydrazine is less clearly defined. First, available 
references for nitrogen solubility in hydrazine give 
different values. It is also unclear how much can be re- 
dissolved. The resultant bubble volume ratio ranges form 
0.72 to i .88*7. Overall, the combined bubble volume 
ratio is somewhere between 2.0 and 3.2%. This yields a 
bubble length of 5.9 to 9.3 cm in the lines between the 
tank isolation valve (Al) and the thruster isolation valves 
(A3 & A4), based on a 290 cm total line length. It is also 
difficult to assess where the bubbles may have collected 
during the filling process, further complicating the 
definition of bubble size and location. Analyses and tests 
described below evaluated the effects of bubble size and 
location to span the probable distributions. 

In January 1991, the B-side manifold, the tank 
crossover manifold (on-orbit refueling coupler, isolation 
valves AIB, A2B, B1A, B2A) and the four propellant 
tanks were loaded with “High Purity Grade" hydrazine 
using a unique propellant loading system developed by 
TRW. The propellant loading system incorporated an air- 
driven positive displacement pump to transfer propellant, 
an electronic mass flowmeter to monitor the rate of 
propellant being loaded, and a reservoir propellant tank to 
remove entrained bubbles before being supplied to the 
propellant pump. 

CGRO was loaded with a total of 1924 kg (4240 Ibm) 
of hydrazine distributed evenly among the four propellant 
tanks. After closing all of the isolation valves in the 
propulsion system, the propellant tanks were then 
pressurized with nitrogen to a flight pressure of 390 psia. 

INITIAL SYSTEM PRIMING EVENT (A SIDE) 

At the completion of the isolation valve leak integrity 
demonstration and final propellant loading and flight 
pressurization, the propulsion system was left in the 
following configuration for launch: 

• Hydrazine loaded to the thruster valves 

• Manifolds between A-side thrusters and isolation 
valves AL, A2. A3 and A4 pressurized to 14.7-15.1 
psia 

• Crossover manifold .'on-orbit refueling coupler. AIB. 
A2B, B l A, B2A) pressurized at 14.7-15.1 psia 
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• Propellant tanks and manifold down to isolation 
. a I s es Al. A3. A IB. and \2B pressurized to 390 
psia 

• B >ide propellant tanks and manifolds loaded and 
pressurized in similar manner as A-side 

• All A-side and B-side isolation valves, including all 
crossover isolation valves, were closed from launch 
through being released from the Shuttle arm. 

On April 7, after being released and establishing the 
required “safe distance” from the Shuttle, activation of 
CGRO propulsion was started. Following nominal pre- 
established planning, the propulsion subsystem activation 
procedure consisted of a series of commands to open A- 
side thruster isolation valves in the following sequence: 
A3, A4, Al, A2. The commands and telemetry responses 
were: 


COMMAND 

TELEMETRY 

OPEN ISO A3 

A3 OPEN 

OPEN ISO A4 

A4 OPEN 

OPEN ISO Al 

i 

• Al CLOSED 

• A1B OPEN 

• Tank Al Pressure Transducer 
Reading Full Scale (510 psia) 


Review of telemetry data confirmed that the anomaly 
had not affected the health of the rest of the spacecraft. A 
decision was made to secure the propulsion subsystem 
and to postpone activation of the subsystem until 1993, 
when orbit raising of CGRO was anticipated to be 
necessary. 

Suspecting that the propulsion subsystem manifold 
may have been subjected to unexpectedly high surge 
pressures, isolation valves Al, A3, A4. A IB were 
commanded closed to minimize the chance of propellant 
leakage from potentially damaged lines and components. 
All valves except crossover A1B were verified to be 
closed following commanding. The fact that A1B did not 
show a response to a valid command indicated either that 
the valve was mechanically damaged and could not 
function, or that the valve's position indicator had been 
damaged and was no longer reliably reporting the valve 
position state. 

A team was formed at the NASA GSFC and TRW- 
Redondo Beach to investigate the anomaly and to assess 
options for safely activating the CGRO propulsion 
system^ 5 *. The results of the investigation were 

distributed to the NASA centers and to the aerospace 
industry. This included dissemination of the anomaly and 
most probable cause via the GIDEP alert system'. 

ANOMALY IMMEDIATE CAUSE 
DETERMINATION 

Initial review of the propulsion system and 
component design data, spacecraft telemetry data, and 


historical precedents quickly concluded that the following 
events had occurred during the priming sequence: 

• Tank isolation valve A l opened as commanded 

• When valve Al opened, rapid propellant flow 
occurred resulting in a pressure surge or 
“waterhammer” 

• The pressure surge overstressed the Tank A l pressure 
transducer sensing element, causing a “zero shift” 
that resulted in an off-scale high reading 

• The pressure transient, flow surge, and/or resulting 
dynamic excitation of the fluid system caused the 
crossover isolation valve (A1B) to change from a 
“closed” to “open” state, and caused the tank 
isolation valve (Al) to change from an “open” to 
“closed” state. 

Although there were no indications of propellant 
leakage through any of the A-side thruster valves, there 
were significant concerns as to the ability of the A-side 
isolation and thruster valves to function properly and 
repeatedly following exposure to high transient pressures. 
Consequently, an extensive program of analytical 
modeling and ground testing was undertaken with the 
objectives of: 

(a) determining whether the CGRO mission should be 
continued on the A-side or B-side, 

(b) determining least-risk method restoring propulsion 
subsystem function, and 

(c) establishing the fundamental (“root”) cause of the 
priming pressure transient. 

SYSTEM ANALYSIS AND MODELING 

A coordinated analysis effort was undertaken at TRW 
to model and understand the dynamic behavior of the 
GRO propellant supply system. In conjunction with the 
analytical effort, experiments were conducted at TRW to 
validate the analysis model. The test setup mimicked the 
flight configuration for critical parameters such as line 
diameter, line lengths, number and location of sharp 
bends, and location of valves. Developmental isolation 
valves of the same design as used on flight were 
incorporated in the setup to insure proper transient 
response to valve opening and closing. Test series were 
conducted using both water and hydrazine. The analysis 
model was exercised to predict the pressure spikes and 
natural frequency response for each test/bubble 
configuration. A key objective of this work was to use the 
empirically-calibrated analysis model of the flight system 
to support decisions on pending activation of CGRO 
propulsion. 

Analysis Model . It is customary to use a lumped 
parameter approach to study the transient behavior of 
liquid flow in a complicated propellant feed system. The 
popularity of such an approach is due to its simplicity and 
numerical efficiency. In the lumped parameter approach. 
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spatial '.anuhon is ignored, resulting m a set ot ordinary 
rather than partial differential equations. These ODE s 
,an he readd> solved by standard integration algorithms, 
such as Runge-Kutta method. 

Various assumptions were made to arrive at the hnal 
analysis model. First, the How was assumed to be 

incompressible and isothermal As a result, the energy 

equation was not considered in the system. This 
assumption was definitely valid because the Mach 
number of the flow {either single phase or two-phase) is 
topically about .001. Another key assumption is that the 
propellant flow was fully turbulent. Again, this 
assumption was valid because the typical Reynolds 
number during the priming event was about 6x10 , well 
exceeding the transition Reynolds number of 2000. 

The lumped parameter approach appears to be sound. 
In a system that consists of only a pipe with high pressure 
at the entrance, it can be shown that closed-form solutions 
for the natural frequency and pressure history exist. 
Furthermore, the peak pressure is predicted to be 
approximately two times the initial pressure difference. 
Such a simple system corresponds to TRW's Series I 
( calibration) tests, in which a tactor of two of initial 
pressure difference was observed. 

For a more complicated system, such as the flight 
subsystem and corresponding ground test simulated 
system, however, numerical analysis (i.e., computerized 
time-marching calculations) is necessary to predict 
pressure levels and transients responses. Due to pressure 
wave interactions arising from complex hydraulic 
configurations, much larger overpressures were calculated 
and measured than are estimated from closed form 
calculations based on simple pipe approximations. The 
results of the detailed modeling are discussed below'. 

Empirical Calibration of Model and Comparison of 
Results . Comparisons between model predictions and test 
data at various transducer locations were made for tests 
involving liquid water and initial pressure differences 
(AP) across the latching isolation valve of 50, 100 and 
375 psid. In the following comparisons, predictions of 
the pressure history were made using pressure drop losses 
across valves based on scaling from specification values, 
and pressure drop losses along the lines according to 
standard Moody diagrams. 

Comparisons between tests and analysis with water 
yielded peak pressure and frequency results of the 
decaying pulses typically within 15 percent of each other. 
This was true for initial APs of 50 psid and 100 psid. The 
actual decay rates of the pressure pulses were quite a bit 
faster than calculated, which is due to a smaller amount of 
damping in the model as compared to the tests. 

Examples of these results are shown in Figures 5 and 
6. It should also be noted that the pressure ratio of peak 
prcs.sure after opening the valve to the pressure 
differential before opening the valve is approximately 5 at 


one location iK*. per Figures 3 and oi in the system and 7 
at another location - k fl , per Figure 3). 

The same tvpes of tests were done with hydrazine. 
The peak pressures were, as expected, slightly higher due 
to the difference in bulk modulus as compared to water 
(see Figures 8 and 9 >. 

In the last test using hydrazine, performed with a AP 
of 375 psid, the A3 valve was closed to prevent too high a 
pressure peak in this part of the system. Note that at 
station K 4 the ratio of peak pressure to initial AP was 5 for 
both analysis and test, as it was for other tests having 
lower initial AP (50 and 100 psid) across the isolation 
valve. 

GROUND TESTS 

The CGRO propulsion subsystem on-orbit anomaly 
investigation included a comprehensive review of 
spacecraft-level ground tests, including electrical circuit 
testing. No credible mechanism was identified that 
related errors in commands or electrical miswiring to the 
telemetry response observed during the A-side activation 
anomaly. 

Supported by the surge pressure analytical model 
(which was calibrated by testing) and by test data using a 
high fidelity mockup of the CGRO propulsion manifold, 
the anomaly investigation team reached the following key 
conclusions 40 6 : 

♦ Larger-than-expected bubble volume was likely left 
in the manifold downstream of thruster isolation 
valves Al, A2, A3, and A4 during propellant loading. 
The primary sources for the large bubble(s) were 
inadequate evacuation of the propulsion system and 
nitrogen gas coming out of solution from the 
hydrazine. The analysis indicated that 2-3.2 percent 
(5.9-9. 3 cm of line length) of the long, large diameter 
manifold volume may have contained bubbles. 

♦ Two key errors were cited as the cause of the large 
trapped bubble volume in the manifold. The first 
error was the use of a low resolution vacuum gauge 
on the propellant loading ground support equipment. 
The second error was the oversight in allowing 
isolation of the manifold at pressure much lower 
(14.7-15.1 psia) than the “pad" pressure (65 psia) ot 
the hydrazine supply tank used in the leak integrity 
demonstration test. 

+ The transient flow model developed by TRW showed 
reasonable correlation to ground test data. It was a 
useful analytical tool to assess pressurization options 
for the CGRO propulsion system. The model 
revealed that the resultant pressure was highly 
sensitivity to the location and size distribution of the 
bubbles. 
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GRO Series I Tests with Water 
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Figure 5. Water/1.36 in 3 Bubble/67 psia Tank 


GRO Series l Tests with Water 
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Figure 6. Water/1.36 in 3 Bubble/117 psia Tank 


GRO Series II Tests with Water 



Figure 7. Water/Multiple Bubbtes/117 psia Tank 


GRO Series IV Tests with Hydrazine 



GRO Series IV Tests with Hydrazine 
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Figure 9. Hydrazine/Multiple Bubbles/392 psia Tank 
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♦ A method referred to vis “fast cycling” ot the isolation 
valve — opening the isolation valves for very short 
duration — was demonstrated to be a viable option to 
safely raise the hydra/me pressure in the manifold to 
design operating pressure. Using breadboard 
electronics and refurbished isolation valves, TRW 
showed that the valve could respond to a command 
sequence consisting ot a valve OPEN command 
\ command to isolation valve '‘open coil"), followed 
"N" ms later with a valve CLOSE command 
{command to isolation valve “close coil"). 

The JSC/White Sands Test Facility (WSTF) played a 
critical role in helping the CGRO anomaly investigation 
team assess risks of the candidate manifold pressurization 
options 5 . WSTF previously had performed extensive 
testing to characterize the likelihood of “adiabatic 
compression decomposition" (ACD) in hydrazine 
propulsion systems. The basic mechanism of ACD is that 
a rapid compression of a gas bubble containing hydrazine 
vapor will heat the gas, which in turn might initiate rapid 
decomposition of the hydrazine vapor (an exothermic 
process), thereby causing peak pressures far in excess of 
the already large waterhammer pressure. 

The objectives of the WSTF tests were: 

1) determine the likelihood of ACD having occurred in 
the initial priming attempt on CGRO A-side 
manifold, 

2) provide additional hydrazine vs. water comparison 
data for TRW's analytical model, and 

3) expose an ACT valve to surge pressures of the levels 
analytically predicted (assuming no ACD) to help 
assess the state-of-heakh of A-side thruster valves 
and the risks of continuing on A-side versus 
attempting priming on B-side. 

The WSTF test setup used simple tubing 
configurations that replicated CGRO manifold line 
lengths and diameters, but no attempt was made to 
replicate individual components, tees, bend angles, etc. A 
fast response, low pressure-drop ball valve was used in 
the test setup to simulate the CGRO isolation valve. This 
test hardware approach permitted many tests to be rapidly 
performed without risk to the higher value, high fidelity 
system mockup used at TRW. 

The WSTF tests provided data for understanding how 
surge pressure was affected by factors such as fluid 
composition (hydrazine versus water), supply pressure, 
bubble size, and tubing diameters and lengths. Conditions 
that trigger significant hydrazine decomposition were 
difficult to define, but the resultant pressures were high 
enough to rupture one of the tubes during the tests. 

While the TRW high fidelity propulsion subsystem 
mockup initially used water for testing, a final series of 
tests was performed using hydrazine. Nineteen hydrazine 


tests were run with low pressures <07 and IP psia) 
upstream and 17 psia downstream of the A l -equivalent 
isolation valve, and there was a final “moment of truth*’ 
test approximating the actual flight conditions with 392 
psia upstream and 17 psia downstream. The resulting 
pressures were high enough to damage the isolation valve: 
it indicated “closed" only after several commands were 
sent to close it, and it had gross leakage in the closed 
position following this test. This test substantiated the 
suspected damage to CGRO isolation valves A1 and A IB 
and indicated the need to switch to B-side and use a 
modified priming sequence to avoid similar damage to 
isolation valve Bi. 

The analytical model, calibrated with empirical data, 
showed that during the on-orbit priming attempt on April 
7 the A-side of the CGRO propulsion subsystem had 
experienced peak surge pressures that ranged from 1,200 
to 4,900 psia. These levels are consistent with analysis of 
pressure loads required to damage the bellows assemblies 
within the isolation valve. 

After assessing the risks and benefits, a decision was 
made to use fast cycling of the isolation valves on the B- 
side manifold. The principal advantage of the B-side 
manifold was that it had not been exposed to the high 
surge pressure. The isolation valve response 
characteristics could be assumed to be reasonably close to 
acceptance test conditions. Further, the “as-launched” 
pressure conditions in the manifold could be modeled 
more accurately in the analysis. 

The fast cycling method was validated at NASA 
GSFC with a CGRO-like command and telemetry system 
and a representative mockup of the CGRO B-side 
propellant manifold. Key hardware in the test setup were 
an engineering model CGRO Electrical Interface 
Assembly (EIA) subdecoder, a breadboard isolation valve 
driver and a refurbished isolation valve. For safety 
reasons, water was used instead of hydrazine. The tests 
characterized the effect on pressurization rate due to 
factors such as valve pulse width (duration between 
commands to the isolation valve open and close coils, or 
effectively the “open time" of the valve), pressure 
downstream of the isolation valve, and spacecraft voltage. 
A linear displacement transducer was used to measure 
movement of the isolation valve poppet as a function of 
electrical pulse width between commands to the open and 
close coils. The tests demonstrated that at 8-12 ms 
(millisecond) open pulse widths, the valve poppet could 
be safely constrained to within the desired 1-8 percent ot 
the full 90 mil (2.3 mm) stroke. 

Results from tests performed at TRW were used to 
determine response differences between the CGRO-like 
command and telemetry simulators used in the tests 
versus the CGRO flight system. The TRW flight 
operations team developed and validated two critical 
procedures that significantly increased the reliability and 
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efficiency of the fast cycling technique. The first 
procedure assured uninterruptible blocks of electrical 
pulse widths to the isolation valve open/close coils, up to 
<i maximum of 14 ms duration, using commands issued 
from the On-Board Computer <OBC). The second 
procedure was a high rate telemetry patch (64 ms 
sampling) tor the pressure transducer. The capability to 
monitor transient pressures during the fast cycling 
operation significantly decreased the time to prime the B- 
side propellant manifold. 

B-SIDE ACTIV ATION & SUBSEQUENT 
OPERATIONS 

The least risk assessment by the anomaly resolution 
team concluded that only the B-side ACT manifold 
should be primed to support the orbit raising operation. 
The ACTs alone could perform the required orbit raising 
without exceeding their qualification limits. Furthermore, 
it was determined that the B-side OAT manifold should 
only be primed just prior to required use of the OATs for 
the controlled re-entry firings at the end of mission life. 

The baseline plan for priming of the B-side ACT 
manifold assumed that the 64 ms/sample pressure 
transducer telemetry patch would not provide sufficient 
sampling of transient pressure data to assess the progress 
of the priming operation. The baseline procedure was as 
follows: 

• Alternately fast cycle isolation valves Bi and B2 for 
500 cycles increasing the valve “open” pulse widths 
in I ms increments from 8 to 12 ms 

• Fast cycle isolation valve B3 for 500 cycles 
increasing the valve “open” pulse widths in 1 ms 
increments from 8 to 12 ms. On the final 12 ms set, 
open B3 for 1 minute 

• Repeat fast cycling sequence for isolation valves Bl, 
B2 and B3 at 300 cycles, 150 cycles, 100 cycles 

• Open isolation valves Bl, B3, and B2 

• Fast cycle cross-over isolation valves B1A and B2A 
tor 100 cycles increasing valve “open” pulse widths 
in l ms increments from 4 to 12 ms 

• Open isolation valves Bl A, B2A. A1B. and A2B. 

Since about ten thousand valve open/close commands 
were anticipated, a 30 second wait was imposed between 
each fast cycle to reduce the thermal stress on the 
electronics. 

While performing the first set of fast cycling with 
isolation valves Bl and B2, the 64 ms sampling of 
pressure telemetry was configured to increase the 
likelihood of capturing the initial cycles of the pressure 
oscillations. Having established this capability, the 
pressure telemetry was used to determine when to stop the 
fast cycling and to command open the isolation valves. 
Implementing this method reduced the time to complete 


the priming operation in half Roughiy 3500 valve fast 
cycles were required to prime the B->ide ACT manifold. 

A second propulsion subsystem on-orbit anomalv 
occurred in May-June 1695 during the calibration burns 
segment of the orbit raising operation, using the 
successfully primed B-side. During the initial test firings, 
one of the four ACTs (“ACT-B2 ) produced unacceptably 
low thrust. 

While several potential causes were investigated, 
evidence led to suspecting the thruster valve, in particular 
the valve s APE-41 I seal area. It was noted that the AVs 
performed with relatively cool thruster start temperatures 
resulted in nominal performance while the AVs initiated 
with warmer start temperatures were anomalous. Review 
of the test history for all the valves in the CGRO lot 

showed a tendency of the valve to change flow 

characteristics with varying temperature and exposure to 
certain fluids. In particular, the acceptance test data show 
increased AP with elevated temperature. Further, it was 
discovered that the valve used on thruster ACT-B2 was 
listed on a SIR (Supplier Information Request) for 
anomalous “non-flow” during acceptance testing at 

elevated temperature. This valve was reworked and later 
accepted. As further evidence, early in the valve 

procurement a valve failed flow- test (no-flow) due to 
elevated temperature and the use of alcohol. Both TRW 
and the vendor investigated the SIR problem and 
concluded the alcohol flow test fluid had caused the seal 
to swell and close off the seat opening. The investigation 
indicated the seal was not sensitive to exposure to 
hydrazine or water at elevated temperature, so it was 
concluded that the valve design was acceptable as long as 
alcohol was not used as the elevated temperature flow test 
fluid. 

Extensive consultations with the thruster valve 
supplier and ground tests conducted at TRW determined 
that the low thrust observed on-orbit was most likely 
related to flexing of the thruster valve seal at high 
propellant mass flow rate. This was confirmed by a hot 
fire test series of sixteen test runs at fixed inlet pressure 
(350 psia) and gradually increasing valve and propellant 
temperature (from ambient to 220 T). For reference, the 
valve temperature reported from the spacecraft during the 
anomaly period was in the range of 95 'F to 120°F. A 
ground spare CGRO thruster was tested, using two 
separate, exchangeable valves that were similar to the 
anomalous flight valve with regard to flow AP as 
measured during valve acceptance test. 

At elevated valve temperatures, the thruster exhibited 
degraded performance for the first 14 seconds of a steady 
state firing, revealing that the seal material caused a flow 
restriction. The initial thruster chamber pressure was 
approximately lO'T of nominal during the first six 
seconds of the run. Thruster temperatures and pressures 
confirmed that this was not a cataKst bed "washout” 
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condition. but rather one ot restricted flow Alter six 
seconds, chamber pressure rose to the normal value and 
continued such throughout the run. No spiking is 
observed after recovery, indicating no excess propellant 
was present in the catalyst bed {i.e.. no washout 
phenomenon). The thruster demonstrated recovery to 
nominal operation upon the arrival ot cooler propellant 
from the feed system. 

To compensate for the low performing ACT-B2 on 
CGRO, two of the four OATs were required to maintain 
attitude control during the orbit raising burns. The B-side 
OAT manifold was successfully pressurized using the fast 
cycling method to open isolation valve B4. 
Approximately 400 valve fast cycles were required to 
prime the B-Side OAT manifold. 

The first orbit reboost of CGRO was completed in 
December 1993. As anticipated, the thrust of ACT-B2 
increased to nominal level as the propellant tank 
"blowdown” pressure decreased with fuel usage. The 
reboost operation consisted of eleven 60-second bums 
and seven 90-second burns, and consumed 1045 lbs (474 
kg) of hydrazine. 

The second orbit reboost was performed March-June 
1997 with nominal performance on all B-side ACTs and 
OATs. The propulsion system also performed [lawlessly 
during the critical controlled re-entry bums that ended the 
CGRO mission on June 4, 2000. 

SUMMARY OF KEY ‘‘LESSONS LEARNED” 

Systems Evacuation and Propellant Loading & Priming 
Three areas were identified where preventive 
measures can be implemented to reduce high pressure 
surge in propulsion systems: surge pressure analysis and 
propellant loading procedure; component design 
evaluation; and system design. 

1. Error : CGRO pressure surge analysis assumption of 
no bubbles in the manifold downstream of the tank 
isolation valve was incorrect. 

• The propellant loading cart gauges did not have 
sufficiently accurate resolution to verify evacuated 
pressures less than 10 torr. 

• In attempting to simplify the propellant loading 
technique to support a special valve leak test, the 
effect of loading supersaturated hydrazine was 
overlooked. 

Preventive Actions : Pressure surge analyses for 

pressurization of the propellant manifold and during 
thruster operation shall include worst-case loading 
assumptions, as dictated by the propellant loading 
equipment evacuation and pressure monitoring 
capabilities. 

There should always be sufficient time allocated 
for stored propellant saturation/desaturation prior to 
loading if large storage pressure changes are expected 


to occur Alternatively, the propellant should be pre- 
conditioned n.c . saturated with the flight pressurant) 
to the state predicted for the on-orbit priming) prior 
to loading into the subsystem. 

3- Error : There was insufficient design evaluation of 

elements in two components that were especially 
vulnerable to high surge pressures. 

• Bellows. Deformation of latch valve bellows 
changes the seating force of the poppet. This change 
in seating force could result in leakage or an inability 
of the poppet to remain open under flow forces. 

• Sensing Diaphragm. Overstress of the diaphragm or 
damage to the electronics of the pressure transducer 
could result in the loss of ability to accurately 
monitor the tank pressure. 

Preventive Actions : The selection of propulsion 

system components shall require demonstrated 
qualification margin with respect to worst case surge 
pressure. Incorporating surge pressure reducing 
devices (see 3. Preventive Actions below) shall be 
considered as a design option to protect sensitive 
component elements. 

3. Error : The CGRO propulsion system design did not 
consider use of surge pressure reducing devices. 
High surge pressure can occur during initial 
pressurization of a low pressure propellant manifold, 
during thruster operation, and during repressurization 
of isolated propellant manifolds (where pressure has 
been reduced by thermally induced back relief; 
propellant leakage from isolated thrusters, etc.) 
Preventive Actions : Components to reduce the rate 
of leakage or to reduce the magnitude of surge 
pressure shall be incorporated in the design, as 
dictated by the surge pressure analyses, component 
qualification margins, and propulsion system mission 
operating plan. Component considerations shall 
include: 

• Series redundant thruster valves offer single fault 
tolerant design and reduce leakage rate in isolated 
manifolds. These features present a stronger 
argument for allowing launching with maximum 
system pressure up to the thruster valves, thereby 
reducing surge pressure concerns associated with 
initial pressurization of low pressure propellant 
manifolds. 

• Pressure surge reducing devices (e.g., oritice. 
cavitating venturi) can be used to protect components 
which are more vulnerable to high transient pressures 
(e.g.. pressure transducers; bellows) 

• High flow resistance bypass valve to slowly prime 
the downstream manifolds before opening the 
isolation valves may be needed for systems with 
long, large diameter lines or tor systems that require 



isolation v>f l he propellant manifolds tor long 
durations 

• Slow opening isolation calves need to be developed 

Parameters Monitored During On-Orbtt Activation 

The key telemetry that was monitored during the 
CGRO on-orbit priming was the propellant manifold 
pressure. With the 64 ms high sample rate capability, the 
operations team was able to capture data indicating 
pressure oscillations during the priming. However, 
incorporating taster response pressure transducers or 
adding capability to sample at higher data rates is not cost 
effective 

Design changes that should be considered include: 

1. Installing the tank pressure transducer on the gas-side 
to protect the sensor trom high surge pressures. 
Note, however, that this defeats the capability to 
resolve rapid transients in the fluid manifolds, which 
is useful in anomaly resolution. 

2. Add pressure transducer downstream of the tank 
isolation valves to assess pressure conditions of 
isolated segments of the manifold. 

Modeli ng as a Problem Resolution Resource & Guide 
The relatively simple one-dimensional, nonsteady, 
lumped parameter (or electric analog) analysis yields peak 
pressure results that are remarkably close to test results. 

Some rather high overpressures can occur during 
initial priming (fill and pressurization) of complex 
propulsion feed systems. It is therefore imperative to do a 
complete non-steady flow analysis and check for 
waterhammer effects during the design of a propellant 
supply system. 

The fluid transient flow model proved to be 
extremely valuable in establishing the anomaly root cause 
and in guiding flight planning and B-side operations. One 
ot its most pivotal contributions was in showing that 
bubble distribution and location were major factors in the 
magnitude of the surge pressure. This was subsequently 
confirmed by tests performed by NASA GSFC. 

For problem resolution, the conditions of bubble size, 
location and distribution are very difficult to define. 
More work is needed to study the transient sensitivity to 
bubbles during priming of an isolated manifold. To 
compensate tor the uncertainty of bubble size and 
distribution, NASA GSFC relies on tests to validate its 
transient flow analyses (e.g., in the design of surge 
pressure suppression orifices). 

At TRW. comprehensive fluid dynamic modeling of 
propulsion systems is performed routinely as part of the 
design effort leading up to a Critical Design Review. 
This modeling parametrically evaluates the effects of fill 
conditions, worst-case bubble loading, system priming 
transients, and thruster-to-thruster hydraulic interactions. 

On the Chandra propulsion system, such modeling 
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identified the need to include surge suppression orifices 
and venturis, and verified that no excessive waterhammer 
pressures would be encountered during any mission 
phase. 

Adequacy Of and Need For A/B Redundancy 

In the case ot CGRO, the .A/B redundant design of 
the propulsion system did increase the mission life and 
enable the safe, controlled re-entry operation. Due to 
increased subsystem complexity and costs, A/B 
redundancy should is most appropriate for missions that 
are high cost (e.g., "Great Observatories’*. EOS), have 
long duration, are a critical "national asset" (e.g., TDRSS, 
GOES, POES, ISS), and are manned. 

Full redundancy of components should also be 
considered for mission critical functions where failure to 
complete the critical function (e.g., mid-course correction, 
orbit insertion) would result in mission failure. In many 
missions, a limited functionally redundant design (as 
opposed to full A/B system redundancy) should be 
acceptable if primary component failure would not result 
in a severely degraded mission. 

Acceptance Data Trend Analysis 

The sensitivity of temperature on the flow 
characteristics of propellant valves utilizing AFE-41 1 seal 
material was recognized early in the CGRO program. 
Flow tests were conducted on test valves with hydrazine 
and water at elevated temperatures, and all flight valves 
were subjected to elevated temperature acceptance tests 
with water. Acceptance criteria were established which 
were believed to be adequate for the CGRO application. 

Following the ACT-B2 on-orbit anomaly, a thorough 
review of the acceptance test data revealed the fact that 
the valve used on thruster B2 exhibited flow 
characteristics within specification but its pressure drop at 
elevated temperature was notably higher than other valves 
in the same lot. This experience underscores the need to 
do a comparative review of the performance of all 
components, as well as a pure specification compliance 
evaluation. In other words, it a lot of components is 
within specification on a particular parameter but one or 
more units is noticeably different from the rest of the 
population, further investigation is warranted to explain 
the difference and its possible impact on the mission. 
Had such an evaluation been performed, the subject valve 
most likely would not have been used for flight. 

Propulsion Testing Adequacy 

Propulsion subsystems are more vulnerable to on- 
orbit anomalies than other spacecraft subsystems due to 
constraints that have been placed on systems-leve! testing. 
Whether the constraints are based on cost, safety hazards 
or risk to the flight hardware, the first-time use of the full 
propulsion subsystem is frequently on-orbit. 
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(in »imd swems-lcvel Jests could have been 
performed on the CGRO propulsion subsystem (or even a 
half-system engineering modeh, including ground support 
equipment, that would have identified the problems that 
contributed to either of the on-orbit anomalies (high surge 
pressure and ACT-B2). For instance, following CGRO, 
the Chandra (AXAF) spacecraft propulsion system was 
mocked up full size and used water as a referee fluid to 
verify that the system priming transients and thruster-to- 
thruster interactions were acceptable prior to flight. This 
testing validated water hammer suppression techniques 
and components that in large part were the result of the 
CGRO anomaly experience. 

Vendor Support and Problem Resolution Teamwork 

The NASA CGRO team had excellent support from 
TRW, organizations within GSFC, sister NASA Centers 
and component vendors. 

The anomaly was fully understood — and the CGRO 
propulsion mission ultimately successful — because 
system and component suppliers had in-house capability 
to do quick turnaround tests in support of critical anomaly 
investigations. Today there appears to be a trend away 
from maintaining such a capability in many aerospace 
businesses. 
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